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At-a-Glance 



TAKE-AWAYS 
Evaluations should: 

• Distinguish carefully between 
gross outcomes (such as how many 
young people attend college after 
high school) and net impacts (the 
extent to which these outcomes 
were caused by the policy or prac- 
tice being studied, rather than by 
other factors). 

• Explore whether policies and prac- 
tices have different net impacts for 
certain subgroups within the high 
school population. 

To understand the prospects for wide- 
spread replication of policies and prac- 
tices that prove successful, the evalua- 
tions should also: 

• Test them on a reasonably large 
scale in a variety of real-world 
settings. 

• Include analyses of program 
implementation and cost. 

Standards and performance indicators 
should take into account the fact that 
certain policies and practices can make 
a significant difference for at-risk stu- 
dents even when the outcomes for 
these students do not appear particular- 
ly favorable in an absolute sense. 
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Evaluating the Impact of Interventions That Promote 
Successful Transitions From High School 

by Michael Bangser of MDRC 

INTRODUCTION 

In today’s information-based economy, it is more important than ever for 
students to attain the advanced skills needed to succeed after high school, 
whether they are continuing their studies, joining the workforce, or juggling 
both school and work. A number of policies and programs have been devel- 
oped to prepare high school students for the transition to postsecondary edu- 
cation and employment. The interventions include, for example, efforts to 
enhance the rigor of high school curricula and to provide counseling, assess- 
ment, financial aid, and other supports to students. This Research Brief 
focuses on the challenges and opportunities presented in evaluating whether 
the interventions achieve their goals of increasing students’ educational 
attainment, employment, and earnings after high school. 1 

THE CONTEXT 

The current wave of high school reform provides a vehicle to conduct rigor- 
ous evaluations of various policies and programs, including those designed 
to prepare students for postsecondary education and employment. The 
accountability that is increasingly required of high schools will be made more 
meaningful by incorporating reliable evidence from these evaluations. 

THE CHALLENGE OF DETERMINING 
EFFECTIVENESS 

Evaluations that address the bottom-line question of the extent to which 
certain interventions increase students’ success in postsecondary education 
and employment face a key challenge: It can be difficult to separate the 
effects of the intervention itself from other factors that might have con- 
tributed to the young people’s experiences after high school. 

The distinction between gross outcomes and net impacts. In determining 
whether interventions are effective, it is important to distinguish between 
gross outcomes (for example, how many young people attend college after 
high school) and net impacts (the extent to which these outcomes were 
caused by the intervention being studied). For example, knowing that 60% 
of the high school students in a particular program went on to college is 
meaningful only if there is a reliable measure (called the counterfactual) of 
how many would have attended college anyway, without participating in 
the program. 
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Determining the net impact of interventions. Some evaluators use large data sets, such as the National Educational 
Longitudinal Study of 1988 (NELS:88), NELS:88 follow-up surveys, the National Longitudinal Survey of Youth 
1997 (NLSY97), and the National Longitudinal Transition Study (NLTS 1 and 2) to explore variations in postsec- 
ondary outcomes for students who were in particular programs or courses of study. Other evaluators try to isolate the 
effects of a program or course of study by comparing the educational and employment outcomes for students who 
participated in it with the outcomes for a matched group of similar students who did not. 

Both approaches need to take into account the extent to which students’ postsecondary outcomes may have been caused 
by factors other than the intervention being studied. Some of these factors can be particularly difficult to measure. For 
example, students who volunteer for a special college preparatory program may have different levels of motivation or 
family support to attend college compared to the students who are used as a comparison group in the research. Later 
differences in college enrollment for the two groups of students could well have been caused by the preexisting differ- 
ences in motivation or family support, rather than by whether the students participated in the program. 

The role of random assignment. The most reliable way to provide a benchmark of how students would have fared in 
the absence of the intervention is widely considered to be random assignment — a lottery-like process in which stu- 
dents, classrooms, or schools are assigned either to a group that receives the intervention being studied or to a control 
group that is precluded from receiving it. (The control group is typically free to participate in any other services 
available in the school and community.) If the assignment process is truly random and the research sample is large 
enough, there should be no systematic preintervention differences between the research groups. Any statistically sig- 
nificant differences in outcomes, therefore, can be confidently attributed to the effects of the intervention. 

The expanded use of random assignment studies in educational settings is an important breakthrough in determining 
the effectiveness of specific interventions. However, this approach presents a number of operational and other chal- 
lenges. For example, random assignment can be an objective and equitable way to allocate scarce program slots when 
the number of eligible and interested students substantially exceeds the available openings; however, the use of ran- 
dom assignment can be criticized for denying services to students if the demand for the program does not substan- 
tially exceed the available openings (Gueron, 2000). Therefore, many evaluations of high school reforms use other, 
quasi-experimental research methods, such as the comparative interrupted time series approach (see, for example, 
Kemple et ah, 2005). 

The discussion below first provides an overview of studies that used a variety of research methods to examine the 
impact of interventions on students’ postsecondary experiences. The focus then turns to five large-scale, random 
assignment evaluations to highlight several key points for policymakers and administrators to consider. 

OVERVIEW OF FINDINGS FROM OUTCOME STUDIES ON 
TRANSITIONS FROM HIGH SCHOOL 

Individual studies and syntheses of the research (primarily reviewing evaluations that did not use random assignment) 
suggest that interventions to improve postsecondary education and employment have produced mixed results. 

The National Educational Longitudinal Study of 1988 (NELS:88). This study followed a cohort of eighth-grade stu- 
dents from 1988 until 2000, when most of them were 26 years old. The study collected extensive information about 
school experiences, educational and occupational aspirations, and postsecondary experiences. Among the findings 
from analyses of the NELS:88 follow-up are that: 

• The academic intensity of the students’ high school curriculum is the most important factor in determining 
momentum toward eventually attaining a bachelor’s degree (Adelman, 2006). 
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• Higher performance scaled to the National Assessment of Educational Progress (NAEP) in mathematics is posi- 
tively correlated with increased enrollment in postsecondary education, the selectivity of the postsecondary insti- 
tutions that students attend, and the likelihood that the students receive a bachelor’s degree (Scott & Ingels, 

2007). 

• Earning at least 20 credits by the end of the first year of college (2- or 4-year) is an important indicator that the 
student is on track to earn a degree. Therefore, beginning the transition process in high school with expanded 
dual enrollment programs that offer postsecondary course work and credits could provide momentum towards 
success in college (Adelman, 2006). 

• Student effort beyond the regular school day is correlated with postsecondary aspirations. For example, almost 
50% of students who spent at least 7 hours per week on homework planned to attend a 4-year institution, 
whereas only 8% of students who spent less than 1 hour per week on homework were planning to do so (Green 
et al., 1995). 

National Assessment of Vocational Education (NAVE). Relying largely on NELS:88 data, NAVE reviewed the impact 
of programs funded by the Carl D. Perkins Vocational and Technical Education Act (Perkins III). The final NAVE 
report, completed under the guidance of an independent advisory panel, concluded that: 

• Vocational courses (now called career and technical education, or CTE) improved later earnings for those stu- 
dents who enrolled in postsecondary education or training (the findings for students who did not pursue further 
education or training were less clear). 

• The earnings gains associated with participation in vocational education were clearest in the short term and to 
some extent in the medium term (defined as 7 years after high school graduation), but it is not clear how much 
these effects persist beyond that point. 

• Although vocational course taking in high school had no effect on overall enrollment in postsecondary educa- 
tion, it was associated with reduced enrollment in 4-year colleges compared to enrollment in 2-year colleges or 
certificate programs. 

• It appears that students’ postsecondary outcomes improve if they take both a vocational program of study and a 
strong academic curriculum (Silverberg et al., 2004). 

Talent Search. The federally funded Talent Search program primarily provides three categories of services: (a) aca- 
demic support (such as test-taking and study skills, as well as advice to students on selecting courses that prepare 
them for college); (b) career development (such as college orientation activities, visits to college campuses, and coun- 
seling); and (c) financial aid (such as individual financial aid counseling, group workshops, assistance with Free 
Application for Federal Student Aid (FAFSA) forms, and scholarship searches). 

Constantine and colleagues (2006) used administrative data to compare outcomes for students who participated in 
Talent Search in three states with outcomes for similar students in the same or other schools. Talent Search partici- 
pants’ enrollment in a public college or university in their state was between 6 and 1 8 percentage points higher than 
that of nonparticipants, depending on the state. (The data collection did not extend to enrollment in other types of 
institutions.) The type of postsecondary institution that hosted the Talent Search project appeared to affect the out- 
comes. Thus, for example, students participating in Talent Search projects hosted by 2-year colleges were more likely 
to enroll in 2-year colleges themselves. 
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The Talent Search study suggests that an effective approach may be assisting low-income students who have college 
aspirations in overcoming information barriers. The evaluators particularly encouraged offering practical information, 
such as direct guidance on how to complete applications for college and financial aid, as well as orienting students to 
what college campuses are like. 

Transition support for students with disabilities. The National Longitudinal Transition Study-2 (www.nlts2.org) is a 
rich source of information on the postsecondary experiences of students with disabilities. Among the findings from 
the study are: 

• While the rate of attending postsecondary education by young people with disabilities has increased, it is still 
only about half that of their peers in the general population, with much of the enrollment for students with dis- 
abilities being in community colleges. 

• Factors that appear to be associated with enrollment in college include students’ having higher functional cogni- 
tive skills, being female, having a better educated head of household, and having progressed to the next grade 
level each year of high school. 

• The patterns of postsecondary experiences differ markedly, depending on the category of disability (Wagner 
et ah, 2006). 

A useful source of information on evidence-based transition practices for students with disabilities can be found on 
the Web site of the National Secondary Transition Technical Assistance Center (www.nsttac.org), which has reviewed 
studies in such areas as student-focused planning (e.g., Individualized Education Program [IEP] development and 
student participation in planning), family involvement, and interagency collaboration. A review of the literature by 
Kohler and Field (2003) concluded that measures such as vocational education, paid work experience, parental 
involvement, and interagency collaboration had a positive impact on student outcomes. An intervention that includ- 
ed coaching youths in the application of student-directed planning skills, peer-based mentorship, and parent support, 
as well as in-service education for school transition staff produced significant increases in students’ involvement in 
transition planning activities and transition awareness (Powers et ah, 2001). 

Other studies. Among the findings from other studies on transitions from high school: 

• Reviewing a number of evaluations of Career Academies (see Table 1 on page 5 for a description of the program 
model), Stern and colleagues (2007) concluded that Academy students outperform non-Academy students on 
various measures of academic success while in high school. Differences in postsecondary education and employ- 
ment were less clear, but where significant differences were found, they favored Academy students. 

• In a study using NLSY97 data, DeLuca and colleagues (2006) found that participation in career-related pro- 
grams does not generally affect college attendance, but taking a higher number of CTE courses compared to 
academic courses is associated with reduced college attendance. 

• Also using NLSY97 data, Neumark and Rothstein (2007) explored the effect of school-to-work programs for 
students who were not likely to attend college in the absence of these programs. The authors concluded that sev- 
eral types of interventions — such as mentoring, co-op programs, and internships and apprenticeships — could 
improve either educational or employment outcomes (or both) for male high school students. The results for 
female students were less clear, although the authors noted that there might be post-high school labor-market 
benefits for females as well. 
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FINDINGS FROM STUDIES USING RANDOM ASSIGNMENT 



In addition to using a random assignment research design, each of the five studies described below: (a) examined an 
intervention that operated in multiple schools or locations, (b) had reasonably large sample sizes, and (c) analyzed 
follow-up data on students’ postsecondary experiences or on high school outcomes that are likely to directly affect 
these postsecondary experiences. The basic elements of the five interventions are presented in Table 1 . 



TABLE 1 

Summary of Interventions Evaluated With a 

Random Assignment Design 2 

Career Academies 

• “School-within-a-school” structure from grades 
9 or 10 through grade 12 

• Combined academic and occupational curricula 
around a career theme, such as health care or 
business and finance 

• Employer partnerships to provide career aware- 
ness activities and work internships related to the 
career theme 

Upward Bound 

• Instruction, tutoring, counseling, and regular 
meetings during the school year, beginning in 
grade 9 or 10, with an emphasis on challenging 
college-preparatory courses 

• An intensive, full-day academic program for 
about 6 weeks each summer, including after 
senior year 

• Cultural activities, college tours, and help with 
applying for financial aid for college 

Career Beginnings 

• Collaboration among a local college (which 
serves as the program sponsor), the public 
schools, and businesses 

• Summer jobs between junior and senior year 

• Workshops and classes during that summer and 
in senior year on topics such as taking college 



entrance examinations and applying for financial 
aid, along with such activities as college and 
career fairs 

• Counseling on educational and career choices 

• Adult mentors serve as role models who actively 
assist youths in planning for the future 

Quantum Opportunity Program 

• Up to 5 years of year-round services, beginning 
in grade 8 

• Comprehensive range of services, including men- 
toring, afterschool tutoring, community service 
activities, computer-based instruction, summer 
jobs, cultural opportunities, and tours of college 
campuses 

• Cash stipends plus deposits in an accrual account 
to be used for postsecondary education or 
approved training programs 

Math-Enhanced Learning in Career and Technical 

Education 

• Extended professional development in which 
mathematics teachers partner with CTE teachers 
to provide ongoing support (but mathematics 
teachers do not teach in the CTE classrooms) 

• Collaborative development of lesson plans 

• Emphasis on the importance of mathematics as a 
workplace skill and identifying opportunities to 
teach mathematics concepts as they naturally 
occur in the curriculum 
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Career Academies. These small learning communities that combine career education and academic content are esti- 
mated to operate in more than 1,500 high schools across the country. A study by Kemple (2004) followed Career 
Academy students (compared to a randomly assigned control group) in nine high schools for 4 years after their 
scheduled high school graduation. Key findings on transitions after high school include: 

• Career Academies produced positive and sustained impacts on a range of labor market outcomes for young men, 
a group that has experienced a severe decline in real earnings in recent years. Through a combination of 
increased wages, hours worked, and employment stability, the young men in the Academy group earned 
$10,000 (18%) more than those in the non-Academy control group over the 4-year follow-up period. 

• Career Academies had no significant effect on the labor market outcomes of young women. The study’s author 
suggests that this may be due, in part, to the fact that the young women in both the Academy and the non- 
Academy groups were more likely than the young men to be attending school or taking care of children. 

• The positive labor market effects were concentrated among Academy group members who were at high or medi- 
um risk of dropping out of high school when they entered the programs. 

• Although nearly 80% of the Academy group enrolled in some form of postsecondary education during the 
4-year follow-up period, this percentage was not higher than the enrollment in postsecondary education by the 
non-Academy group. It is noteworthy, however, that the improved labor-market outcomes for young men did 
not reduce their pursuit of postsecondary education during the follow-up period compared to the control group. 

Upward Bound. Upward Bound is a longstanding, federally funded program that provides academic instruction and 
other supports to prepare low-income students for college. It currently operates in more than 700 locations around 
rhe country. An impact evaluation conducted by Myers and colleagues (2004) included a nationally representative 
sample of 67 Upward Bound projects hosted by 2- and 4-year colleges. Based on follow-up data covering a period 
when the majority of the research sample had been out of high school for about 2 years, the evaluators found that: 

• Almost three-fourths of students in both the Upward Bound group and the control group attended some form 
of postsecondary education. Upward Bound had no effect on overall enrollment or total credits earned, but it 
may have increased the proportion who enrolled in 4-year colleges. 

• Upward Bound more than doubled (from 18% to 38%) the eventual enrollment in 4-year colleges for students 
who had lower expectations of attending such a college when they applied to the Upward Bound program in 
grade 9 or 10. There was no effect on 4-year college enrollment for students with higher educational expecta- 
tions when they applied to the program. 

• Upward Bound also at least doubled the number of credits earned at 4-year colleges by students who initially 
had lower educational expectations. Again, there was no effect for those with higher educational expectations. 

Career Beginnings. This program model is aimed at high school juniors from low-income families who have shown 
perseverance in their schoolwork but are not among the top students in their class. In an evaluation of programs in 
seven urban areas, follow-up data through roughly 1 year after scheduled high school graduation showed that: 

• Even though members of the control group received more services and had higher college attendance than 
anticipated, Career Beginnings led to increased college attendance. During the follow-up year, 53.2% of the 
Career Beginnings group attended a 2- or 4-year college, an increase of 4.7 percentage points over the figure 
for the control group. 
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• This increase in college attendance was, however, accompanied by decreased enrollment in vocational schools. 
Therefore, participation in Career Beginnings caused a shift in the type of postsecondary education rather than 
an increase in the overall rate of enrollment in postsecondary education. 

• The Career Beginnings group initially worked less and earned less than the control group, but this earnings dif- 
ferential disappeared by the end of the follow-up period. 

• The effect on college attendance varied considerably among the seven sites, but no clear story emerged about dif- 
ferential program effectiveness for subgroups such as young men and young women (Cave & Quint, 1990). 

Quantum Opportunity Program ( QOP). Operating in seven cities, QOP offered students in the lower two-thirds of 
their eighth-grade class up to 5 years of comprehensive, year-round services. In addition, for every hour of participa- 
tion, QOP enrollees earned both a cash stipend and a deposit in an accrual account to be used for postsecondary 
education or approved training programs. 

Nearly 1,100 QOP-eligible students were randomly assigned to either the program or the control group. The 
research sample was followed for nearly 5 years after their scheduled high school graduation date. The evaluators 
noted that the seven sites varied considerably in their implementation of the QOP model, with enrollees attending 
fewer program activities than expected. With respect to the program’s effects, the study found that: 

• QOP did not increase the enrollees’ likelihood of graduating from high school with a diploma or GED. 

• Although there had been an indication of increased entry into postsecondary education or training when QOP 
enrollees were in their late teens and early twenties, this effect was not sustained as the research sample entered 
their mid-twenties. For those sample members who did enroll in postsecondary education or training, the even- 
tual attainment of postsecondary credentials was no greater for the QOP group than for the control group. 

• At the time of the final follow-up, when the members of the research sample were entering their mid-twenties, 
QOP enrollment had not led to increased employment or earnings (Schirm et ah, 2006). 

Math-enhanced learning in career and technical education classes. In this study, career and technical education 
(CTE) teachers who volunteered to participate were randomly assigned to one of two groups: The first group was 
paired with mathematics teachers in a community of practice to develop CTE instructional activities that integrated 
more mathematics into the occupational curriculum; the second group taught their CTE classes as usual. Stone and 
colleagues (2006) found that, after 1 year of math-enhanced CTE lessons averaging 10% of class time: 

• Students in math-in-CTE classrooms showed “moderate to medium” improvements, compared to students in the 
control classrooms, on standardized tests of mathematical achievement. 

• The additional time spent on mathematics in these classrooms did not detract from students’ acquisition of tech- 
nical skills and knowledge in occupational areas. 

• Students in the math-in-CTE classrooms had higher scores on a widely used college placement test, suggesting 
that they might be less in need of remedial courses at the postsecondary level. 

INTERPRETATION AND IMPLICATIONS OF THE FINDINGS 

A number of important lessons can be drawn from these random assignment evaluations: 

The effect of other available services. In each of the studies, the members of the control group did not receive the 
particular intervention being tested, but they remained eligible for other services in their schools and communities. 
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Especially if applicants to a program are highly motivated — as was the case, for example, in Career Beginnings — the 
control group can be expected to find and use these other services. 

The net impact findings should therefore be interpreted as the incremental impact of the interventions over and 
above the mix of alternative available services. The evaluations do not assess the effects of the interventions compared 
to a no-service control group. 3 The studies suggest the need for program planners to consider precisely how a new 
offering will fit within and enhance the range of services that are already available, especially for motivated students. 

The size of the impacts. The net impacts found in rigorous evaluations are often more modest than policymakers and 
administrators have come to expect from the operating data that they typically see on program outcomes. This is 
because the operating data include what participants would have achieved even without the additional program serv- 
ices; as discussed earlier, the net impact of the program takes this into account by focusing on the difference between 
the experiences of the program enrollees and the experiences of the randomly assigned control group. 

However, impacts of the size found on certain outcomes in these random assignment studies still provide a solid 
foundation on which to build. Technical assistance, professional development, and curriculum development can help 
sustain and extend these positive impacts (Grubb & Stern, 2007). For example, the Career Academies Support 
Network offers comprehensive support and staff development services to strengthen the operation of “small learning 
communities” and Career Academies. Similarly, the promising results of the math-in-CTE study have led the 
National Center for Career and Technical Education to offer assistance to states and schools interested in effective 
replication of the model. 

Importance of analyzing subgroup results, including for students at risk and students with disabilities. The net 
impacts can differ for specific groups of students. For example, the Career Academies study found positive impacts 
on post-high school labor market outcomes for males but not for females. Moreover, the net impacts of interventions 
(i.e., the difference they make) are sometimes larger (or found only) for more at-risk students. This was the case for 
the labor market experiences of likely dropouts in the Career Academies study, as well as for enrollment in 4-year 
colleges by Upward Bound participants who began with low expectations of obtaining a bachelor’s degree. 4 

The larger net impacts for more at-risk groups can occur even if their gross outcomes do not appear to be particularly 
favorable. For example. Upward Bound participants who initially had low expectations of obtaining a bachelor’s degree 
were less likely to go on to 4-year colleges than those with higher initial expectations (38% compared with 56%). 
However, the Upward Bound program made no significant difference for the students with higher expectations, 
while it more than doubled the 4-year college-going rates for students who started with lower expectations. 

Implications for targeting of programs and operating indicators. The Career Academies and Upward Bound findings, 
in particular, suggest that it can be important for programs to reach out to serve at-risk students. To the extent possi- 
ble, however, these students should be included in programs that serve students with a range of abilities; otherwise, 
teachers and students might have lower expectations of the programs, seeing them as weaker, remedial efforts 
(Kemple, 2004; Moore & Myers, 2004). 

The standards and performance indicators used to measure the success of programs need to take into account the 
lower outcomes that can result from serving larger numbers of at-risk students, where the net impact of the program 
may nevertheless be substantial (Moore & Myers, 2004). Unless standards and performance indicators are adjusted, 
they could inadvertently underestimate the effectiveness of the programs serving at-risk students and inappropriately 
reward programs simply for serving higher performing students. 
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CONCLUSION 

Policymakers, administrators, and researchers need to work together to determine which interventions are most effec- 
tive in preparing students for post-high school transitions. Expanded use of random assignment studies should con- 
tinue to the extent feasible, along with efforts to strengthen alternative, quasi-experimental methodologies. Net 
impact studies should be conducted at scale in real-world settings and be accompanied by analyses of program imple- 
mentation and costs. This will provide a better understanding of the prospects for widespread replication of success- 
ful efforts. 

Conducting these evaluations poses considerable challenges, but it can also improve high school accountability. In 
particular, incorporating the results of rigorous evaluations will help ensure that standards and performance indicators 
reward those who make a real difference for the students they serve. 
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END NOTES 

1 See also the companion Issue Brief, Bangser, M. (2008). Preparing high school students for successful transitions to postsec- 
ondary education and employment. Washington, DC: American Institutes for Research, National High School Center. The 
Issue Brief offers program design and implementation lessons for policymakers and administrators to consider. 

2 This table is based on the evaluators’ descriptions of the programs at the time of the study; the programs’ current opera- 
tions might differ. 

3 The net impacts can also be affected by the fact that some of the students who are randomly assigned to receive the inter- 
vention either do not participate at all, or they participate for a relatively short time or less intensively than the program 
design envisioned (Kemple, 2004; Schirm et ah, 2006). 

4 A similar result was found in the Structured Training and Employment Transitional Services (STETS) demonstration, 
which did not focus specifically on transitions from high school but did provide employment-related services to 1 8- to 
24-year-old young adults with mental retardation. STETS increased the regular, unsubsidized employment rates for par- 
ticipants with moderate or mild retardation, but it had no impact on regular job holding for participants in the borderline 
range (Kerachsky et ah, 1985). 




